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TEXT MINING METHOD AND APPARATUS FOR EXTRACTING 
FEATURES OF DOCUMENTS 

CROSS-REFERENCE TO RELATED APPLICATIONS 
[01] This application claims priority to Japanese Patent Application No. 2000-197421 
filed on June 29, 2000, the contents of which is incorporated herein by specific reference. 

BACKGROUND OF THE INVENTION 

1 . FIELD OF THE INVENTION 

[02] The present invention relates to a text mining method and apparatus for 
extracting features of documents. In particular, the invention relates to a text mining 
method and apparatus for extracting features of documents, wherein features are 
extracted such that all mutually associated documents and terms are placed near each 
other in the feature space. Applications of the invention include document and/or web 
retrieval, associated term retrieval, document classification. 

2. DESCRIPTION OF THE RELATED ART 

[03] In text mining as a technology for squeezing desired knowledge or 
information by making analysis of text data, effective feature extraction of documents is 
an important task for efficiently performing document and/or web retrieval, associated 
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term retrieval, document classification and so on. As a typical document feature 
extracting method, a vector-space model as set out on page 313 of "Automatic Text 
Processing" (Addison- Wesley, 1989) is frequently used. 

[04] In the vector-space model, when terms selected as indices in the documents, 
namely index terms representing the contents of the documents, are t in number, a vector 
Vi is used respectively to correspond to an index term Ti to define a t-dimensional vector 
space. All vectors forming thus the defined vector space can be expressed as a linear 
combination of t in number of the vectors corresponding to t in number of the index 
terms. In this vector space, a document D r is expressed as follows: 

D , = £ x „ v , (i) 

[05] In the foregoing expression (1), x ir active on Vi is the contribution of the index 
term Tj to the document D r and represents a feature of the document. The feature is an 
amount representing the term frequency of the index term in the document. A vector [x r i, 
x r2 , ... Xrt]' of t x 1 (t rows and one column) becomes a feature vector of the document D r . 
As the simplest case, when the index term Tj appears in the document D r , x ir is set to 1. 
When the index term Tj does not appear in the document D r , x ir is set to 0. In a more 
complicated case, as set forth in the foregoing publication on page 279 to 280, two 
quantities are used. These two quantities are a term frequency tf ri of the index term Tj in 



-2- 



DocketNo. 14321.31 



7 



the document D r and a document frequency df\ of documents containing the index term Ti 
in all documents registered in the document database. 

[06] For the group of documents consisting of d in number of documents, a t x d 
term-document matrix X can be defined as follows: 

X=[xi,x 2 , ...,x<i] 

[07] Here, a t-dimensional vector xj = [xji, Xj 2? ..., x jt ]' expresses the feature vector of 
the document Dj, and '(dash) represents matrix inversion. 

[08] Fig. 1 is an illustration showing one example of documents, translated from 
Japanese sentences, registered in a document database, where "ronin" is a romanized 
word meaning students who, having failed a school entrance-exam of a particular 
IS academic year, are preparing for one next year. Fig. 2 is an illustration showing one 

example of a term-document matrix taking the Kanji (Chinese) characters appearing on 
the documents shown in Fig. 1 as index terms. Kanji terms are underlined in Fig. 1. In 
Fig. 2, among a character string "let me know about" appearing in all of the documents 1 

jo z: 

" 2§ £ § § s to 3, the Kanji term "know" is checked off from the index terms. Fig. 3 is an illustration 

& b ^ H w < 

W S H W H H 

a o < £ K d 

g ^ | o I g showing one example of an actual input question, translated from Japanese, from a user, 

^ g S 3 V3 t> 

2 - w O < w ^ 

^ w h lij ^ ? 

where Kanji terms are underlined. If the index terms of Fig. 2 are used to express the 
question, the question can be expressed with the term-document matrix shown in Fig. 4. 



y\ H w < -5 
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[09] 



In general, when the vector-space model is used, similarity sim (D r , D s ) of two 



documents D r and D s can be expressed as follows: 



t 



sim{D r ,D s ) = 




(2) 



[010] When the similarity of the question and each document of Fig. 1 is judged on 
the basis of the meaning of the question of Fig. 3, the question of Fig. 3 is the most similar 
to the document 3 of Fig. 1 . However, using the feature vectors as shown in Figs. 2 and 4, 
the similarity of each document of Fig. 1 and the question of Fig. 3 is respectively 
sim(document 1, question) = 0.5477, sim(document 2, question) = 0.5477, sim(document 
3, question) = 0.5477. In short, all have the same similarity. 

[011] As a solution for such a problem, a method called Latent Semantic Analysis 
(LSA) was proposed in "Journal of the American Society for Information Science" 1990, 
Vol. 41, No. 6, pp. 391 to 407. This method extracts latent meaning of the documents on 
the basis of co-occurrences of the terms and is significantly outstanding in terms of 
retrieving efficiency. Here, "co-occurrences of terms" represents a situation where the 
terms appear simultaneously in the same documents/statements. 
[012] The LSA extracts a latent semantic structure of the documents by performing 
singular value decomposition (SVD) for the term-document matrix. In the obtained 
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feature space, mutually associated documents and terms are located near each other. In a 
report placed in "Behavior Research Methods Instruments & Computers" (1991), Vol. 
23, No. 2, pp. 229 to 236, retrieval using the LSA indicates a result of 30% higher 
efficiency in comparison with the vector-space model. LSA will be explained hereinafter 
in more detail. 

[013] In LSA, at first, singular value decomposition is performed for the t x d 

term-document matrix X as set out below, 
jg X-ToSoDo' (3) 

\f_ [014] Here, To represents an orthogonal matrix of t x m, So represents a square 

}y diagonal matrix ofmxm with taking m in number of the singular values as the diagonal 

elements and setting 0 to the other elements. Do' represents an orthogonal matrix of m x 
^ d. In addition, let us assume that 0 > d > t, and arrange the orthogonal elements of So in 

descending order. 

[015] Furthermore, in LSA, with respect to the feature vector x q of t x 1 of a 

3 o z: 

" i £ I £™ document D q , the following conversion is performed to derive a LSA feature vector y q of 

* O < o I 5 

^ a J H « s 

idSSgjf n xl: 

'tS v~> H jjj -< < 

y q = s- ! rx q (4) 



2 % < a o H 
2 ^ "* 3 



[016] Here, S is a square diagonal matrix of n x n taking the first to (n)th of the 
diagonal elements of So, and T is a matrix of t * n drawing the first to (n)th columns of To- 
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Pi % *J 

m 

<tJ H 
5 W H 

2 < 



o 5 p 

won 

,j w U 

O H 

< w W 

w < < 

o W J 

£ ° H 

2 n 

< 



[017] As an example, results of singular value decomposition of the term-document 
matrix shown in Fig. 2 are given below. The matrices To, So and Do are expressed as 
follows: 



T - 



0.1787 


-0.3162 


0.3393 


0.1787 


-0.3162 


0.3393 


0.1787 


-0.3162 


0.3393 


0.4314 


-0.3162 


-0.1405 


0.4314 


-0.3162 


-0.1405 


0.1787 


0.3162 


0.3393 


0.1787 


0.3162 


0.3393 


0.4314 


0.3162 


-0.1405 


0.4314 


0.3162 


-0.1405 


0.1787 


0.3162 


0.3393 


0.2527 


0.0000 


0.4798 



s n = 



Do = 



2.7979 0 0 
0 2.2361 0 
0 0 1.4736 



0.5000 -0.7071 0.5000 
0.5000 0.7071 0.5000 
0.7071 0.0000 -0.7071 



[018] Let us assume that the dimension t of the LSA feature vectors is 2 and 
applying the foregoing expression (4) to each feature vector of the term-document matrix 
in Fig. 2. Then, the LSA feature vectors of the documents 1, 2 and 3 are respectively 
[0.5000, -0.7071]', [0.5000, 0.7071]' and [0.7071, 0.0000]'. In addition, applying the 
foregoing expression (4) to the feature vector of Fig. 4, the LSA feature vector of the 
question from the user becomes [0.6542, 0]'. 
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[019] Applying the foregoing expression (2) to the LSA feature vectors obtained as 
set forth above, the similarity of the question of Fig. 3 and each document of Fig. 1, 
become respectively, sim(document 1, question) = 0.5774, sim(document 2, question) = 
0.5774, and sim(document 3, question) = 1.0000. Thus, a result that the document 3 has 
the highest similarity to the question can be obtained. Considering a help system 
application or the like utilizing computer networks, an answer statement of the document 
3 registered in the document database will be returned to the user who asked the question 
of Fig. 3. 

[020] For singular value decomposition, an algorithm proposed in "Matrix 
Computations 1 ', The Johns Hopkins University Press, 1996, pp. 455 to 457, is frequently 
used. In the report of "Journal of the American Society for Information Science" set forth 
above, there is a statement that the value of the number of rows (or columns) n of the 
square matrix S is preferably about 50 to 150. In addition, in the foregoing report of 
"Behavior Research Methods, Instruments, & Computers", it has been indicated that 
better efficiency can be attained by pre-processing using the term frequency or document 
frequency instead of simply setting each element of the feature vector to 0 or 1 before 
performing LSA. 

[021] However, in the algorithm for singular value decomposition proposed in the 
foregoing "Matrix Computations", memory space in the order of the square of the number 

- 7 - Docket No. 14321.31 



of index terms t (t 2 ) is required at the minimum. This is because a matrix of t x t is 
utilized for bidiagonalization of a matrix in the process of calculation of basis vectors 
spanning a feature space from a given term-document matrix. The prior art is therefore 
not applicable to document database holding a very large number of terms and data. 
Furthermore, the prior art requires complicated operations of matrices irrespective of the 
number of data. 
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SUMMARY OF THE INVENTION 
[022] The present invention has been worked out in view of the problems set forth 
above. In a first aspect of the present invention, a text mining method is provided for 
extracting features of documents using a term-document matrix consisting of vectors 
corresponding to index terms representing the contents of the documents. In the 
term-document matrix, contributions of the index terms to each document act on 
respective elements of the term-document matrix. The method comprises: 

a basis vector calculating step of calculating a basis vector spanning a 
feature space, in which mutually associated documents and terms are located in 
proximity with each other, based on a steepest descent method minimizing a cost; 

a feature extracting step of calculating a parameter for normalizing 
features using the term-document matrix and the basis vector, and extracting the 
features on the basis of the parameter; and 

a term-document matrix updating step of updating the term-document 
matrix to a difference between the term-document matrix, to which the basis 
vector is not applied, and the term-document matrix, to which the basis vector is 
applied. 

[023] In a second aspect of the present invention, a text mining method is provided 
for extracting features of documents wherein the cost is defined as a second-order cost of 
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the difference between the term-document matrix, to which the basis vector is not 
applied, and the term-document matrix, to which the basis vector is applied. 
[024] In a third aspect of the present invention, a text mining method is provided for 
extracting features of documents wherein the basis vector calculating step comprises: 

an initializing step of initializing a value of the basis vector; 

a basis vector updating step of updating the value of the basis vector; 

a variation degree calculating step of calculating a variation degree of the 
; ^ value of the basis vector; 

U a judging step of making a judgment whether a repetition process is to be 

=U terminated or not using the variation of the basis vector; and 

)% a counting step of counting the number of times of the repetition process. 

H [025] In a fourth aspect of the present invention, a text mining method is provided 

for extracting features of documents wherein the basis vector updating step updates the 
basis vector using a current value of the basis vector, the term-document matrix and an 

53 2 - 

£ jj ^ 1 3 s updating ratio controlling the updating degree of the basis vector. 

g < o % s 
& j h a < 

O g < H X ° 

§ ^ > o B g [026] In a fifth aspect of the present invention, a text mining method is provided for 

£ g § 3 § 5 

2 " « O < w fc^ 
< l3 H W 

sS*!!^ extracting features of documents wherein when all basis vectors and normalizing 
parameters required in extracting the features have been already obtained, the calculation 
of the normalizing parameters in the basis vector calculating step and the execution of the 
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feature extracting step are omitted. In addition, the feature extracting step extracts the 
features using the basis vectors and the normalizing parameters that have been already 
obtained. 

[027] In a sixth aspect of the present invention, a text mining apparatus is provided 
for extracting features of documents using a term-document matrix consisting of vectors 
corresponding to index terms representing the contents of the documents. In the 
term-document matrix, contributions of the index terms to each document act on 
respective elements of the term-document matrix. The apparatus comprises: 

basis vector calculating means for calculating a basis vector spanning a 

feature space, in which mutually associated documents and terms are located in 

proximity with each other, based on a steepest descent method minimizing a cost; 
feature extracting means for calculating a parameter for normalizing 

features using the term-document matrix and the basis vector, and extracting the 

features on the basis of the parameter; and 

term-document matrix updating means for updating the term-document 

matrix to a difference between the term-document matrix, to which the basis 

vector is not applied, and the term-document matrix, to which the basis vector is 

applied. 
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[028] In a seventh aspect of the present invention, a text mining apparatus is 

provided for extracting features wherein the cost is defined as a second-order cost of the 

difference between the term-document matrix, to which the basis vector is not applied, 

and the term-document matrix, to which the basis vector is applied. 

[029] In an eighth aspect of the present invention, a text mining apparatus is 

provided for extracting features of documents wherein the basis vector calculating means 

comprises: 

initializing means for initializing a value of the basis vector; 

basis vector updating means for updating the value of the basis vector; 

variation degree calculating means for calculating a variation degree of 
the value of the basis vector; 

judging means for making a judgment whether a repetition process is to 
be terminated or not using the variation of the basis vector; and 

counting means for counting the number of times of the repetition 
process. 

[030] In a ninth aspect of the present invention, a text mining apparatus is provided 
for extracting features of documents wherein the basis vector updating means updates the 
basis vector using a current value of the basis vector, the term-document matrix and an 
updating ratio controlling the updating degree of the basis vector. 
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[031] In a tenth aspect of the present invention, a text mining apparatus is provided 
for extracting features of documents wherein when all basis vectors and normalizing 
parameters required in extracting the features have been already obtained, the calculation 
of the normalizing parameters in the basis vector calculating means and the execution of 
the feature extracting means are omitted. In addition, the feature extracting means 
extracts the features using the basis vectors and the normalizing parameters that have 
been already obtained. 

[032] In an eleventh aspect of the present invention, a computer program product is 
provided for being executed in a text mining apparatus for extracting features of 
documents using a term-document matrix consisting of vectors corresponding to index 
terms representing the contents of the documents. In the term-document matrix, 
contributions of the index terms act on respective elements of the term-document matrix. 
The computer program product comprises: 

a basis vector calculating step of calculating a basis vector spanning a 

feature space, in which mutually associated documents and terms are located in 

proximity with each other, based on a steepest descent method minimizing a cost; 
a feature extracting step of calculating a parameter for normalizing 

features using the term-document matrix and the basis vector, and extracting the 

features on the basis of the parameter; and 
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a term-document matrix updating step of updating the term-document 
matrix to a difference between the term-document matrix, to which the basis 
vector is not applied, and the term-document matrix, to which the basis vector is 
applied. 

[033] The feature extracting apparatus disclosed in this specification is constructed 
by defining a cost as a second-order function of a difference between the term-document 
matrix, to which the basis vector is not applied, and the term-document matrix, to which 
IS the basis vector is applied. The apparatus merely requires the following means: 

H a. basis vector calculating means for calculating a basis vector 

lit applying a steepest descent method to the cost; 

b. feature extracting means for calculating a parameter for 
|f normalizing features using the term-document matrix and the basis vector, and 

extracting the features on the basis of the parameter; 

c. term-document matrix updating means for updating the 

>* >7 

gj <§ ^ § 3 i term-document matrix to the difference between the term-document matrix, to 

O O < £ x D 

%^>oBt which the basis vector is not applied, and the term-document matrix, to which the 

z 2 23 o h w 
^ k o < m 5 
5 c« s- 1 uj < < 

I S < i g £ basis vector is applied, in order to prevent redundant extraction of features; and 

O 3 w 

d. feature extraction control means for controlling execution of 
respective means. 
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[034] The basis vector calculating means repeats calculation on the basis of the input 
term-document matrix to finally derive one basis vector. The repetition process is 
terminated when the variation degree of the basis vector becomes less than or equal to a 
predetermined reference value. 

[035] The feature extracting means calculates a parameter for normalizing the 
features on the basis of the input basis vector and the term-document matrix, and extracts 
a feature for each document. 

[036] The term-document matrix updating means updates the term-document matrix 
on the basis of the input basis vector. 

[037] The feature extraction control means repeats execution of each means until the 
number of the features defined by the user is satisfied. When the basis vectors and the 
normalizing parameters have been already calculated, execution of the basis vector 
calculating means and calculation of the normalizing parameters in the feature extracting 
means are omitted. Then, the feature extraction can be performed with the construction 
incorporating the already obtained basis vectors and the normalizing parameters. 
[038] According to the present invention, a text mining method for extracting 
features of documents using a term-document matrix consisting of vectors corresponding 
to index terms representing the contents of the documents, wherein contributions of the 
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index terms act on respective elements of the term-document matrix, comprises the 
following steps: 

i. a basis vector calculating step of calculating a basis vector 
spanning a feature space, wherein mutually associated documents and terms are 
located in proximity with each other based on a steepest descent method 
minimizing a cost; 

iL a feature extracting step of calculating a parameter for normalizing 
features using the term-document matrix and the basis vector, and extracting the 
features on the basis of the parameter; 

iii. a term-document matrix updating step of updating the 
term-document matrix to a difference between the term-document matrix, to 
which the basis vector is not applied, and the term-document matrix, to which the 
basis vector is applied; and 

iv. a feature extraction control step of controlling execution of 
respective steps. 

[039] Therefore, concerning feature extraction of documents in text mining, the 
features having the same nature as those obtained by LSA can be extracted with smaller 
memory space than the apparatus or method executing LSA. In addition, specific 
software or hardware for extracting the features can be easily implemented. 
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[040] The above and other objects, characteristics and advantages of different 
embodiments of the present invention will become more apparent from the following 
descriptions of embodiments thereof taken in conjunction with the accompanying 
drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[041] Fig. 1 is an illustration showing one example of documents registered in a 
document database; 

[042] Fig. 2 is an illustration showing one example of a term-document matrix with 

taking Kanji terms appearing in the documents shown in Fig. 1 as index terms; 

[043] Fig. 3 is an illustration showing one example of a question actually input by a 

user; 

[044] Fig. 4 is an illustration showing a term-document matrix obtained from the 
question in Fig. 3; 

[045] Fig. 5 is an illustration showing one embodiment of a feature extracting 
apparatus according to the present invention; 

[046] Fig. 6 is an illustration showing one example of a hardware construction for 
implementing the present invention; 

[047] Fig. 7 is an illustration showing a structure of a term-document matrix data 
fde; 

[048] Fig. 8 is an illustration showing a structure of a basis vector data file, wherein 
the calculated basis vectors are stored; 

[049] Fig. 9 is an illustration showing a structure of a feature data file; 
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[050] Fig. 10 is an illustration showing a structure of a normalizing parameter data 
file; 

[051] Fig. 1 1 is a flowchart showing calculation of a basis vector in basis vector 
calculating means; and 

[052] Fig. 12 is an illustration showing one example of an automatic document 
classifying system employing one embodiment of the feature extracting apparatus 
according to the present invention. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[053] Fig. 5 is an illustration showing one embodiment of a feature extracting 
apparatus according to the present invention. As shown in Fig. 5, feature extraction 
control means 200 has term-document matrix updating means 210, basis vector 
calculating means 220, feature extracting means 230. The reference numeral 100 denotes 
a term-document matrix data file, 300 denotes a basis vector data file, 400 denotes a 
feature data file, 450 denotes a normalizing parameter data file. In the term-document 
•y matrix data file 100, a term-document matrix of collected document data is stored. The 

H term-document matrix updating means 210 reads the term-document matrix from the 

til term-document matrix data file 100, and transfers the read term-document matrix to the 

J ; basis vector calculating means 220 and the feature extracting means 230 without updating 

U the term-document matrix, in a first iteration process. 

[054] In a second and subsequent iteration processes, the terms-document matrix is 
updated on the basis of the basis vector transferred from the basis vector calculating 

pq e ^ 

8 <§ ^ I 3 s means 220. The result of updating is transferred to the basis vector calculating means 220 

* o < o I g 

05 & J H a < 

%3>ooh and the feature extracting means 230. The basis vector calculating means 220 calculates 

S 5 e w o n 

W P < ^ 

!§<§Sh one basis vector through a repetition process based on the term-document matrix 



transferred from the term-document matrix updating means 210. Then, the degree of 
variation of the basis vector in respective repetition is monitored for terminating the 
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repetition process when the degree of variation becomes less than or equal to a 
predetermined reference value. The basis vector calculating means 220 stores the 
calculated basis vector in the basis vector data file 300 and in conjunction therewith, 
transfers the calculated basis vector to the term-document matrix updating means 210 and 
the feature extraction means 230. The feature extracting means 230 extracts one feature 
of each document on the basis of the term-document matrix transferred from the 
term-document matrix updating means 210 and the basis vector transferred from the basis 
:~ vector calculating means 220. The result is stored in the feature data file 400, and also the 

^ parameter for normalizing the features is stored in the normalizing parameter data file 

TU 450. 

\% [055] Execution of the term-document matrix updating means 210, the basis vector 

^ calculating means 220 and the feature extracting means 230 is taken as one iteration 

process. Number of times of iteration processes will be indicated by suffix i, and number 
of features designated by the user is indicated by suffix n. The feature extraction control 

So ^ 

8 <j ^ 1 3 s means 200 repeats the process until a condition, i = n is satisfied. On the other hand, in a 

^ 9 < o I ffi 

g | g g g g 

| ^ 5 o d £ case where all of the required basis vectors and the required normalizing parameters have 

E ^ £ 3 ^ s 

Z OT 2 < °2 *4 
5 CO H OJ < < 

5 S < I * ^ already been obtained, execution of the basis vector calculation means 220 and 

calculation of the normalizing parameters in the feature extracting means 230 may be 
omitted. Therefore, in such a case, the feature extraction control means 200 may be 
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constructed with the term-document matrix updating means 210 incorporating the known 
basis vectors and normalizing parameters, and with the feature extracting means 230. 
[056] Fig. 6 is an illustration showing one example of a hardware construction for 
implementing the present invention. As shown in Fig. 6, the feature extracting apparatus 
includes the following components: 

a central processing unit (CPU) 10 performing control for the overall 
apparatus, 

^iJ a memory 20 for storing the program and providing a temporary data 

H storage region required for executing the program, 

\U a keyboard 30 for inputting data, and 

a display 40 for generating a display screen. 

^ [057] The programs to be executed by the feature extraction control means 200, the 

term-document matrix data file 100, the basis vector data file 300, the feature data file 
400, and the normalizing parameter data file 450 are stored in the memory 20. 

>- ^ 

3§ = 

s H £ I ^ s [058] By taking this construction, the feature extraction is performed by CPU 10 

g^SoS^ receiving the command from the user through the keyboard 30, a mouse pointing a 

E § § 3 § ^ 

^ O o h W 

•zf O < oo ^ 
% oo H uq < <J 

|g<|S^ desired position on the display 40, or the like. It should be noted that, in the example 
shown in Fig. 5, the feature extraction control means 200 has a stand-alone construction. 
However, the feature extraction control means 200 may be built-in other systems. 
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[059] Fig. 7 is an illustration showing a structure of the term-document matrix data 
file. In Fig. 7, the reference numerals 101-1, 101-2, 101 -d correspond to t-dimensional 
term-document data A consisting of d in number of data. Here, X = [xi, X2, ... Xd], *j = 
[xji, Xj 2? ... Xj t ]' are defined to express the term-document data A with a t x d matrix X. 
[060] Fig. 8 is an illustration showing a structure of the basis vector data file storing 
the calculated basis vectors. In Fig. 8, the reference numerals 301-1, 301-2, ... 301-n 
correspond to t-dimensional basis vector data B consisting of n in number of data. The 
(i)th element 301-i corresponds to an output value of the basis vector calculating means 
220 in the (i)th iteration process in Fig. 5. In the following disclosure, this element is 
expressed by a t x 1 column vector Wi = [wn, w i2 , w lt ]\ 

[061] Fig. 9 is an illustration showing a structure of the feature data file. In Fig. 9, 
the reference numerals 401-1, 401-2, 401-n correspond to d-dimensional feature data 
C consisting of n in number of data. The (i)th element 401-i corresponds to an output 
value of the feature by the feature extraction means 230 in the (i)th iteration process in 
Fig. 5. This element is expressed by an 1 xd row vector yi = [yn, y\i, yid]- 
[062] Fig. 10 is an illustration showing a structure of the normalizing parameter data 
file. In Fig. 10, the reference numerals 45 1-1 ,45 1-2, 45 1-n correspond to normalizing 
parameter data D consisting of n in number of data. The (i)th element 45 1-i corresponds 

- 23 - Docket No. 14321.31 



to an output value of the normalizing parameter by the feature extracting means 230 in the 
(i)th iteration process in Fig. 5. 

[063] Using the foregoing definitions, an implementation of feature extraction in the 
shown embodiment will be explained. The term-document matrix updating means 210 
reads out X from the term-document matrix data file 100 only when i = 1, namely in the 
first iteration process, to store in a t x d matrix E without performing any arithmetic 
operation. Accordingly, E = [e b e 2 , e d ], ej = [eji, ej 2 , ej t ] f = [xji, Xj 2 , Xj t ]'. In order 
to prevent redundant extraction of the features extracted in the preceding iteration 
processes, E is updated in the (i)th iteration using the current value and the basis vector 
calculated in the immediately preceding iteration process. The result of updating is 
transferred to the basis vector calculating means 220. A value of E in the (i)th iteration, 
E(i), will be expressed by the following expression (5): 



[064] Here, E(i) = [ei(i), e 2 (i), e<j(i)] 5 each element ej(i) of E(i) is defined by ej(i) = 
[eji(i), ej 2 (i), ej t (i)]\ Namely, when i > 2, the term-document matrix is updated to a 
difference derived by subtracting the term-document matrix, to which the basis vector is 
applied, from the term-document matrix, to which the basis vector is not applied. 
[065] Fig. 1 1 is a flowchart showing calculation of the basis vector in the basis 
vector calculating means. In Fig. 1 1, a value of Wj in the (k)th repetition is expressed by 




(5) 
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Wi(k) = [wii(k), Wi2(k), Wit(k)] 1 . At first, at step S500, the suffix k is initialized to L 
Subsequently, the process is advanced to step S510 to initialize respective element of 
Wi(l) with an arbitrary value between -C to C. Here, the value of C may be a positive 
small value, such as C = 0.01. At step S520, in order to calculate the basis vector 
spanning a feature space where mutually associated documents and terms are located in 
proximity with each other, a second-order cost expressed by the following expression (6) 
is provided. 

~t E(e*.(0-w,yj 2 (6) 

let m=x i = i 

[066] Here, "terms are placed in proximity" means that the positions of the terms are 
close with each other within a feature space, and "documents are placed in proximity" 
means that the positions of terms included in respective documents are close in the 
feature space. On the other hand, a cost means an object to be minimized. In the shown 
embodiment, the cost is defined as a second-order function of the difference between the 
term-document matrix, to which the basis vector is not applied, and the term-document 
matrix, to which the basis vector is applied, as expressed by the expression (6). Here, 



p/ hJ f_H < 

S § S? w P fc 
52 R < h x p 

£ 3 e w o n 7i 

20u!0[hW ^ im 

5 ^ h S < ? 

2 < O o (_ 

§ 85 ~ 5 is the (m)th element of a 1 x d vector < y l which is defined as follows: 



y, =[^1, y l2 > -Yid] = Wi f E(0 (7) 
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So = not on the basis of the value of 5 i(k). As a result of the judgment, if termination is 

* p <c o § 5 

& g J H S < 

o o 5 b £ 5 determined, the process is advanced to step S560, and otherwise, the process is advanced 

Z 2 ^ O H W r 

| i | i « < to step S550. Here, in Fig. 11, £ i is a positive small value, such as j3 \ = 1 x 10 . 

3 fe < © o h 

* [071] At step S550, the value of the counter k is incremented by 1 . Then, the process 
is returned to step S520. At step S560, Wi is stored as the (i)th data of the basis vector data 



[067] For the cost, the steepest descent method is applied to update the value of Wj as 
expressed by the following expression (8). 

w i (*+l) = w i (t)+^(E(/)-w i (*)z i (*)) z.,(ky (8) 

a 

[068] Here, \i i(k) is an update ratio controlling the degree of updating in the (k)th 
repetition, which is initialized by a positive small value when k is 1, such as /i i(l) = 0. L 
Every time of increment of k, the value is decreased gradually. In the alternative, it is also 
possible to set the value at a constant value irrespective of k. On the other hand, Zj(k) is 
defined as follows: 

z i (k) = w i (k) , E(i) (9) 
[069] At step S530, 5 i(k) indicating the degree of variation of Wi is derived as 
follows: 



$m = JS(w 7f (^+i)-w 7; w) 2 (io) 



J= l 



[070] At step S540, a judgment is made whether the process is to be terminated or 
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file 300. At the same time, Wi is transferred to the term-document matrix updating means 
210 and the feature extracting means 230. In the feature extracting means 230, the feature 
yi and the normalizing parameter p { are calculated in the following manner. 

y-i - y i / Pi (ii) 

[072] Here, p\ is defined as follows: 




[073] The feature y\ and the normalizing parameter p\ are stored respectively in the 
feature data file 400 and the normalizing parameter data file 450 as the (i)th data. 
[074] Fig. 12 is an illustration showing one example of an automatic document 
classifying system employing the shown embodiment of the feature extracting apparatus. 
In Fig. 12, the reference numeral 601 denotes term-document matrix calculating means, 
602 denotes classifying means. The classifying means 602 may be implemented by a 
method disclosed in "Journal of Intelligent and Fuzzy Systems", published on 1993, Vol. 
l,No. 1, Pages 1 to 25. 

[075] The document data stored in the document database E is taken in the automatic 
document classifying system 600. In the automatic document classifying system 600, a 
term-document matrix is derived in the term-document matrix calculating means 601. 
The result of calculation of the term-document matrix is transferred to the feature 
extraction control means 200. The feature extraction control means 200 extracts the 
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features from the received term-document matrix. The extracted result is output to the 
classifying means 602. In the classifying means 602, the result of classification is output 
on the basis of the input features. 

[076] To evaluate the present invention, feature extraction of actual document data 
related to an entrance examination system was performed. It has been confirmed that the 
present invention could extract the features of the same nature as those extracted using 
the conventional LSA. 

[077] Next, concerning the size of the memory space, in a typically practical case 
where the number of terms t is significantly greater than number of documents d (t » d), 
the conventional LSA requires in the order of t of the memory size, the present invention 
merely requires the memory size in the order of t*d for calculating respective basis 
vectors. Furthermore, in order to realize the prior art, a complicated matrix operation 
apparatus is required. The system according to the invention, however, can be easily 
realized with an apparatus that performs simple arithmetic operations. Namely, 
according to the present invention, the LSA feature extraction can be performed using a 
smaller memory space and a simpler program. In addition, this simple program may be 
loaded in a digital signal processor (DSP). Therefore, a specific chip for feature 
extraction can be produced easily. 
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[078] Hereinafter, the results of respective means executing the shown embodiment 
of the feature extracting apparatus for the documents of Fig. 1 and the question of Fig. 3 
will be shown. 

[079] A. Documents of Fig. 1 

[080] First, let X denote the term-document matrix of Fig. 2. 

[081] I. First Iteration in Feature Extraction Control Means 200 (i = 1) 

[082] According to the foregoing expression (5), the term-document matrix 

updating means 210 outputs E(l) expressed by the following expression to the basis 

s 220 and the feature extracting means 230. 

0" 
0 
0 
1 
1 
0 
0 

1 
1 

0 

1_ 

vector calculating means 220, initialization is performed with 
• wi(l) at [0.0100, -0.0100, 0.0100, -0.0100, 0.0100, -0.0100, 
, -0.0100, 0.0100]', 1 at a fixed value 0.1, 0 i at 1 x 10 -6 . The 
calculation shown in Fig. 1 1 is performed for a hundred thirty-two times. Then, the basis 



vector calculating mean: 




E(l) 
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[083] 



In the basis 



setting the basis vectoi 



0.0100, -0.0100, 0.0100 
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vector wi = [0.1787, 0.1787, 0.1787, 0.4314, 0.4314, 0.1787, 0.1787, 0.4314, 0.4314, 
0.1787, 0.2527]'is stored in the basis vector data file 300, and output to the feature 
extracting means 230 and the term-document matrix updating means 210. 
[084] * First Repetition in Basis Vector Calculating Means 220 (k = 1) 
[085] From the foregoing expression (8), 

wi(2) = [0.0103 -0.0097, 0.0103, 0.0093, 0.0107, -0.0103, 0.0097, -0.0100, 0.0100, 
-0.0103, 0.0103]' 

wi(2) - wi(l) = 10" 3 x [0.3332, 0.3334, 0.3332, 0.6668, 0.6666, -0.3332, -0.3334, 0.0001, 
-0.0001,-0.3332, 0.3332]' 

5 i(l) = 0.0103 

[086] * Second Repetition in Basis Vector Calculating Means 220 (k = 2) 
[087] From the foregoing expression (8), 

Wl (3) = [0.0107, -0.0093, 0.0107, -0.0085, 0.0115, -0.0107, 0.0093, -0.0100, 0.0100, 
-0.0107, 0.0107]' 

wi(3) - wi(2) = 10 3 x [0.41 10, 0.41 12, 0.41 10, 0.8001, 0.7998, -0.3665, -0.3668, 0.0224, 
0.0221,-0.3665,0.3887]' 

6 j(2) = 0.0015 
[088] - syncopated ~ 
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[089] * A Hundred Thirty-Second Repetition in Basis Vector Calculating Means 
220 (k= 132) 

[090] From the foregoing expression (8), 

wi(133) - [0.1787, 0.1787, 0.1787, 0.4314, 0.4314, 0.1787, 0.1787, 0.4314, 0.4314, 
0.1787, 0.2527]' 

wi(133) - wi(132) = 10" 6 x [-0.3020, -0.3020, -0.3020, -0.3020, -0.3020, 0.3020, 0.3020, 
0.3020, 0.3020, 0.3020, 0.0000]' 

8 i(132) = 9.5500 x 10' 7 
[091] In the feature extracting means 230, the operations shown in the expressions 
(11) and (12) are performed for outputting: 

yi =[0.5000,0.5000,0.7071] 

and 

pi = 2.7979 

to the feature data file 400 and the normalizing parameter data file 450. 
[092] II. Second Iteration in Feature Extraction Control Means 200 (i = 2) 
[093] In the term-document matrix updating means 210, from the foregoing 
expression (5), E(2) expressed as follows is output to the basis vector calculating means 
220 and the feature extracting means 230: 
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E(2) = 



0.7500 
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0.7500 


-0.2500 


-0.3536 


0.7500 
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-0.3536 
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0.3964 
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0.1465 
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0.7500 
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-0.3536 


-0.3535 


0.5000 



[094] In the basis vector calculating means 220, initialization is performed with 
setting the basis vector w 2 (l) at [0.0100, -0.0100, 0.0100, -0.0100, 0.0100, -0.0100, 
0.0100, -0.0100, 0.0100, -0.0100, 0.0100]', ix 2 at a fixed value 0.1, J3 2 at 1 * 10" 6 . The 
calculation shown in Fig. 1 1 is performed for a hundred nineteen times. Then, the basis 
vector w 2 = [0.3162, 0.3162, 0.3162, 0.3162, 0.3162, -0.3162, -0.3162, -0.3162, -0.3162, 
-0.3162, 0.0000]' is stored in the basis vector data file 300, and output to the feature 
extracting means 230 and the term-document matrix updating means 210. 
[095] * First Repetition in Basis Vector Calculating Means 220 (k = 1) 
[096] From the foregoing expression (8), 

w 2 (2) = [0.0102, -0.0098, 0.0102, -0.0096, 0.0104, -0.0105, 0.0095, -0.0103, 0.0097, 
-0.0105,0.0102]' 

w 2 (2) - w 2 (l) = 10" 3 x [0.2154, 0.2156, 0.2154, 0.3822, 0.3821, -0.451 1, -0.4513, -0.2844, 
-0.2846, -0.4511,0.1666]' 
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6 2 (1) = 0.0011 



[097] * Second Repetition in Basis Vvector Calculating Means 220 (k = 2) 
[098] From the foregoing expression (8), 

w 2 (3) = [0.0105, -0.0095, 0.0105, -0.0092, 0.0108, -0.0110, 0.0090, -0.0106, 0.0094, 
-0.0110, 0.0103]' 

w 2 (3) - w 2 (2) = 10" 3 x [0.2624, 0.2626, 0.2624, 0.4413, 0.441 1, -0.5152, -0.5154, -0.3364, 
-0.3366, -0.5152, 0.1786]' 

8 2 (2) = 0.0013 
[099] — syncopated — 

[0100] * A Hundred Nineteenth Repetition in Basis Vector Calculating Means 220 (k 
= 119) 

[0101] From the foregoing expression (8), 

w 2 (120) = [0.3162, 0.3162, 0.3162, 0.3162, 0.3162, -0.3162, -0.3162, -0.3162, -0.3162, 
0.0000]' 

w 2 (120) - w 2 (119) = 10' 6 x [0.3327, 0.3333, 0.3327, -0.1375, -0.1381, 0.3332, 0.3326, 
-0.1377, -0.1383, 0.3332, -0.4712]' 

5 2 (119) = 9.8141 x 10" 7 
[0102] In the feature extracting means 230, the operations shown in the expressions (1 1) 
and (12) are performed for outputting: 
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y 2 -[0.7071, -07071,-0.0000] 

and 

p 2 = 2.2361 

to the feature data file 400 and the normalizing parameter data file 450. 
[0103] From the results set forth above, the feature vectors of the documents 1, 2 and 
3 in Fig. 1 are respectively [0.5000, 0.7071] r , [0.5000, -0.7071], [0.7071, -0.0000]. 
Comparing these with the features of the LSA of respective documents shown in the 
explanation of the prior art, the second element of each vector is of opposite sign but has 
the same absolute value. Accordingly, concerning calculation of similarity in the 
expression (2), they have the same nature as the features of LSA. 
[0104] B. Question of Fig. 3 

[0105] Here, let us use the basis vectors stored in the basis vector data file 300 and the 
normalizing parameters stored in the normalizing parameter data file 450 during 
extraction of the features of the documents of Fig. 1 . Thereby, execution of the basis 
vector calculating means 220 and calculation of the normalizing parameter in the feature 
extracting means 230 are omitted. Let X denote the term-document matrix of Fig. 4. 
[0106] I. First Iteration in Feature Extracting Means 200 (i = 1) 
[0107] In the term-document matrix updating means 210, E(l) expressed as follows 
from the foregoing expression (5) is output to the feature extracting means 230. 
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0 
0 



E(l) = 



0 

1 

0 
0 



[0108] In the feature extracting means 230, arithmetic operation according to the 
foregoing expressions (11) and (12) is performed using the feature vector wi and the 
normalizing parameter p\ obtained upon extraction of the features of the documents of 



Fig. 1 to output 



yi = [0.6542] 



to the feature data file 400. 



[0109] II. Second Iteration in Feature Extraction Control Means 200 (i = 2) 
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[0110] In the term-document matrix updating means 210, using the feature vector wi 
obtained upon performing feature extraction of the documents shown in Fig. 1, from the 
foregoing equation (5), E(2) expressed as follows is output to the feature extracting 



means 230. 



-35- 



DocketNo. 14321.31 



E(2) = 



-0.3271 
-0.3271 
0.6729 
0.2103 
0.2103 
0.6729 
0.6729 
-0.7897 
0.2103 
-0.3271 
-0.4626 



[0111] In the feature extracting means 230, arithmetic operation according to the 
foregoing expressions (11) and (12) is performed using the feature vector W2 and the 
normalizing parameter p 2 obtained upon extraction of the features of the documents of 



Fig. 1 to output 



y 2 = [-0.0000] 



to the feature data file 400. 



[0112] From the result set forth above, the feature vector of the question of Fig. 3 
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becomes [0.6542, -0.0000]', comparing the value explained in the prior art, the second 



element has the same absolute value. 



[0113] The present invention has been described in detail with respect to preferred 
embodiments. It will now be apparent from the foregoing to those skilled in the art that 
changes and modifications may be made without departing from the invention in its 
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broader aspect. It is the intention, therefore, in the apparent claims to cover all such 
changes and modifications as fall within the true spirit of the invention. 
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